Between Mediation and Making, CIMSp: A Technoètic Modus Operandi

The following paper describes an ongoing research project whose goal is to define a scalable, hybrid production and deployment protocol (CIMSp) for the creation of virtual environments (VE). Ultimately, the aim is to establish a creative workflow and infrastructure that embodies architectural and urban design activity as practiced by the research unit. The objective of the present paper is to schematically outline the current state of the research and its practical and theoretical context for further development. A theoretical position will be stated which assumes that the content, tool, epistemological, and speculative realms are consubstantial (technoèsis). The practical endeavour is to create the informational and embodied temporal--spatial condition of possibility for the imaginative production of cultural artifacts. It must accommodate varying individual and collaborative forms and styles of making and no presumption of a self-enclosed and referential system is made. A critical position is particularly compelling when this production is immersed in technological modalities of making where information and embodiment are inextricably intertwined. CIMSp is based on the workflow from acquisition and creation to output and storage. The work environment is comprised of a select set of software applications and visualization technologies. Secondly, an XML-based content and information management system is under construction to ensure project quality control, rigorous documentation practices, and bi-directional knowledge feedback procedures to enable an effective and resource-full workflow. Lastly, scalability of output modalities for use in the design process and for final presentation from WWW deployment to a high-resolution collaborative work environment (CWE) is being developed. The protocol is a multi-user mode of creation and production that aims to transform the technologies and their interrelation, thus dramatically impacting the creative process and intended content. It is a digital production workflow that embodies intensive visualization criteria demanded by the end users. The theoretical and practical intention of CIMSp is to provisionally structure the collaborative creative process and enable a choreographed movement between the realms of the technologically mediated and made in the pursuit of significant digital content creation. Introduction: team and the content-based project There is no single solution for the 3D digital work environment, and any presumption of such would impede innovation and unwontedly systematize the design process. The Carleton Immersive Media Studio production protocol (CIMSp) is first and foremost a synthetic creative process that aims to strike a balance between technological mediation and the imaginative requirements of the users. The objective of the research is to establish a protocol for the creative workflow and production of 3D imaging and modeling of the built environment in order to make delivery, collection, and creation consistent in terms of qualitative and quantitative measures. The definition of “protocol” in the Oxford Concise Dictionary (1995) is: 1a official, esp. diplomatic, formality and etiquette observed on state occasions, etc. b. the rules, formalities, etc of any procedure, group, etc. 2 the original draft of a diplomatic document.” The sense to be invoked is that of diplomacy, etiquette and general conviviality in the context of negotiation and agreement. The CIMSp is set in this context of diplomacy and negotiation between mediation and making rather than that of a strictly instrumental or rigid procedure. The research agenda of the interdisciplinary group assumes an intertwining of project-based research and pure research with expertise contributed from university and public and private sector strategic partners. The host research unit is the Carleton Immersive Media Studio (CIMS) located within the School of Architecture at Carleton University (Ottawa, Canada). The mandate of CIMS is the production of relevant cultural content that concurrently transforms and impacts the development and usage of new technologies in a variety of cultural practices. CIMS researchers are broadly interdisciplinary and strategic partners from cultural institutions as well as agencies and industry contribute expertise and resources. The team explores and investigates the creation of virtual and built environments with new media technologies as they transform and are transformed by our perceptual and epistemological worlds. The premise is that cultural content creation will greatly determine and be determined by technological innovation and advance in applied and pure research. Kockelkoren (2003) uses the term technoèsis to describe the intertwining of (technologically) mediated experience and the conferral of meaning. “Technoèsis stresses the mediated character of every act of conferring and understanding meaning.” He goes on to say that “The formation of artistic images and technological design evolve in parallel. Images, technologies and even scientific theories emerge in a social process, and that process is not marked by monocausality. That is why I prefer to speak of the co-evolution of technological design, the formation of images and the conferral of meaning – in other words, technoèsis.” The immediate objective is to digitally re-construct and capture a high resolution, accurate, and interactive model of a historically significant set of buildings and urban area of Ottawa for rhetorical and design speculations. In addition to creating significant cultural digital artifacts, the production of this content will exemplify the workflow while simultaneously testing, developing, and integrating the given technology. The primary buildings to be digitally reconstructed are the old Ottawa Union Station (now the Government Conference Centre (GCC)) and the Chapel of the Convent of Our Lady of the Sacred Heart (a.k.a. Rideau Chapel). The fate of the GCC is currently under review by the federal and municipal government and digital documentation is desired for use in the design and decision making process. It is presumed that a sophisticated historically situated architectural and urban presentation would enable a more effective design and policy consideration. The Rideau Chapel raises a different set of issues related to the value of heritage and architectural artifacts. The interior of the Rideau Chapel was dismantled in 1972 when the convent was razed and subsequently re-assembled in the National Gallery of Canada. Now a museum artifact, it will be laser scanned and the former structure and context will be reconstructed and digitally modeled from photographs and drawings. Both projects are significantly underway and the bulk of major data gathering (including 3D laser scans) will be complete by the fall 2004. Assets will be used throughout the project in presentations and design exercises utilizing collaborative work spaces and immersive environments. At the time of this paper submission, the primary buildings and larger site context has been modeled in two 3D applications and an image modeling software. These assets are being used for further modeling, compositing and baking of high resolution textures in an entertainment industry 3D animation application. The next stage of the project is to utilize non-contact measurement techniques (3D laser imaging) to capture interior spaces and details that would be impossible and/or time consuming to model and where eliminating the negative aspects of an increased visual fidelity (file size, cost, time) is justified. The project will include a rich media presentation of the Rideau Chapel that imaginatively and critically situates it in its historic, conservation, and museum context. It will be done in collaboration with the National Gallery of Canada and a multimedia exhibit in the Chapel space is tentatively planned. The exhibit/installation will utilize immersive virtual environments within the actual physical space. In addition, the project-based research on the Union Station and Rideau Chapel will develop new media technologies and be used to test the interactivity of the model in a virtual reality environment in collaboration with high performance visualization cluster research conducted with IBM Researchers from the T.J. Watson Lab. The project will serve in the technological advance and development of a prototype 3D laser camera and image modeling techniques created at the National Research Council of Canada. Lastly, and most importantly for the discussion at hand, the material will be used to determine the workflow for the seamless integration of various digital files with digitally constructed models, image modeled components, and 3D laser scans. As such, the production of the various digital artifacts will serve as the basis for the development of CIMSp in a multi-modal, multipurpose environment. Tools and Techniques: acquisition, production, output The intent of the current research is to minimize the various applications necessary in order to effectively and efficiently construct and present virtual environments (VE) of existing or lost buildings. Issues such as speed, accuracy, cost, flexibility, scalability, and desired output modes must be weighed. Concerns in the making of a VE include: • data collection to cover all necessary details and material qualities; • degree of visual fidelity (high resolution compositing, rendering, lighting); • complex and high geometric accuracy and efficiency of model size • pre-visualization and maintenance of a visual fidelity at a real-time rate; • real-time interactivity (design and presentation); • multimodal output from immersive output (VR) to WWW deployment • scalability of output, visual fidelity, asset quality, and infrastructure. A set of software applications have been chosen and the integration, interoperability, and functionality are being developed in the context of a collaborative work environment (CWE) where resources, expertise, and knowledge are shared. A number of factors were considered in choosing these primary software applications in the workflow including: inherent capabilities; interoperability with other applications; capability to customize interface and functionality (scripting); learning curve; support and training; existing community of users; access to research and/or development partners; short and long-term cost; vision and future development; social-cultural context of users and company. It must be stated that there is no technically objective way to select certain applications over others for such a workflow. Decisions regarding technologies to be implemented are set within a distinct social and cultural context within which, the product, company, and culture surrounding the product operate and envision itself. The decision to use the software as discussed below recognizes this ultimately subjective position. Although several software packages were tested, no single “best” solution is thought to exist and a ‘family’ of applications was strategically assembled. While a socially and technically informed decision, this somewhat alchemic process is preferred over the endless technological trap of attempting to find the ultimate technology. An even less desirable situation is, if budget permits, to defer the decision and opt for a plethora of like applications assuming a neutrality of media whose usage is simply an issue of individual preference. Thus, three primary software applications are being used that encompass the acquisition (ShapeCapture), production (Maya/MentalRay), and multimodal output (Virtools) of 3D digital assets. ShapeCapture: image-based 3D acquisition, modeling, rendering: The 3D digital construction of existing physical environments is typically a user demanding, labor intensive process of digitally modeling by manually positioning elements. Although desired during the design phase of a project, there are several drawbacks to this approach when modeling an existing context. “First, the process is extremely labor intensive, typically involving surveying the site, locating and digitizing architectural plans (if available), or converting existing CAD data (again, if available). Second, it is difficult to verify whether the resulting model is accurate. Most disappointing, though, is that the renderings of the resulting models are noticeably computer generated; even those that employ liberal texture mapping generally fail to resemble real photographs” (Debevec, 1996). Image-based modeling and image-based rendering (photogrammetry) can expedite the process of acquiring 3D data and textures through technological automation. This non-contact, imagebased technique mediates the recovery of geometry from a series of photographs and is well suited to architectural and urban environments as it exploits the rectilinear and perspectival nature of the built world. The benefits of photogrammetry are the fact that is a cost effective, efficient, non-intrusive technique that is capable of recording precise geometric and texture data from the level of overall context to details and engineering analysis. Lastly, the photographs can be used for archival purposes (El-Hakim, 2003). Maya and MentalRay: 3D modeling, rendering, lighting, animation: The most critical application to select for the workflow was the 3D modeling, rendering, and animation software. The production phase is the moment in the project when the positive creative slippage occurs and mediation can work to the detriment of the imaginative process. Thus, the software has to be robust, flexible, customizable, and well suited to the architectural imagination. Although Alias Maya is biased toward character animation, the underlying modeling, rendering, and animation capabilities make it the most powerful application for our purposes. Maya and MentalRay are sophisticated tools for the construction of models and their rendering, texturing, lighting, and animation. Photo-realism is not always desired by any means but the technical requirements of matching photo-realistic visual fidelity and real-time rendering and interaction sets the hardware and software standard for production and presentation. Maya is capable of sophisticated rendering and animation while MentalRay dramatically extends the rendering and lighting capabilities of the package. Maya Embedded Language (MEL) is an open source scripting language that allows the customization of the interface and functionality of the program. MEL is a scripting language that provides the foundation for Maya operation. MEL commands and scripts can be used to create a number of changes and refinements to the operation and functionality of the application. Further, it is used to transform the user interface. CIMS is developing an Architectural Edition of Maya in partnership with Alias. Virtools: real-time interactivity, behaviour, deployment: Virtools is an online and off-line development and authoring application for programming behaviour and interactivity to objects and environments. Virtools offers a number of packages for the creation, publication, and multimodal output of digital assests for the creative process and final presentation. As such, it accommodates a scalable workflow and collaborative production environment. It has project management capability that integrates with other assets management solutions. The scripting language (VSL) and drag and drop behaviours operate within an intuitive interface. The SDK allows the creation of custom behaviour that can be deployed offline, over the web, or in an immersive environment. It contains a native Behaviour Engine and Render Engine for real-time visualization which support 3D modeling objects and animation in Maya, 3DS Max, and Lightwave. An Immersive Player and Web Player are used in the multi-modal deployment (VR, WWW, rich media) of the assets for design and presentation. Hybrid Method: technological biases and resistances The proposed hybrid method combines both geometrybased and image-based techniques to achieve a balance between effectiveness and creativity in the acquisition, documentation, and design process. A creative process is only as good as the positive resistances and efficiencies that it presents to the creator as an oscillation between the realm of technological mediation and the creative, embodied, user-dependent realm of making. A strategy of choreographing these resistances inherent in the materiality and biases of the media used is a necessary position to take. A balance between the mediated and the made is one that is gained through proper knowledge and familiarity of craft in general, the interrelation of the various technologies involved, and each specific medium. A keen awareness of the crafting of digital artifacts sets up criteria for identifying (often intuitively) preferred resistances and biases inherent in each application. A balance between the technologically mediated operations of certain software application and the making of a digital model or image first recognizes the inherent biases of the medium in its specificity and then accepts (or rejects) the resistance certain operations submit to the creator. The ability to customize functionality or the space of the interface is a great advantage. For instance, the choice of Maya was initially problematic due to the fact that the given interface is not well suited to an architectural way of conceiving/visualizing virtual and real worlds. However, Maya Embedded Language allows the user to combine actions and to customize the interface. Each application has a materiality and offers resistances to the creative process that must not be taken for granted. Another example is the widely varying sophistication and quality of 3D packages to render and light an environment. The lighting and rendering capabilities of MentalRay are powerful and capable of a greater degree of fine tuning, subtlety, and mimicry than 3DS Max or RenderZone. On the other hand, the limitations of RenderZone require the user to utilize applications such as Photoshop or AfterEffects to craft the image in a way that is highly stylized where the hand of the creator is evident. Combining applications to achieve a desired effect is often more desirable in the design process precisely due to the struggle to overcome technological resistances. The integration of heterogeneous application files is inevitably an issue of translation between applications rather than a transcriptive, technologically mediated action. CIMSp: oscillation between the mediated and the made CIMSp has been conceived with a critical and imaginative notion of a dynamic and heterogeneous technoètic relation between the mediated and the made. “Under the present conditions of technoètic pluralism, people are exposed to heterogeneous forms of disciplining. People are free to regret the fact if they so choose, but it also entails opportunities for unprecedented freedom (Kockelkoren, 2003). The protocol and its circuitous infrastructure are spatial and temporal conditions of possibility. It attempts to do this on a number of levels. First, an attempt is made to choreograph the process by reciprocally responding to and structuring the artifactual resistances of the workflow components. Secondly, the hybrid scenario is supported by the CIMSp infrastructure that implements an XML-based two-way documentation and feedback mechanism for the various realms of information and knowledge surrounding any given project. The dynamic inter-relation between the mediated and the made is fundamentally what the CIMSp is targeting. On one hand, what is at issue is the streamlining and making more effective the CWE by requesting and archiving the necessary project and heuristic information. On the other, what is at issue is allowing for the vague ambiguity and inefficiency of the creative act specific to the digital craft. As such, it must inherently recognize the provisional nature of the form and style of making at an individual and group level. Taken seriously are the concerns of a number of critics that bemoan the loss of embodied knowledge, spontaneous creativity and the heterogeneous field of making that is presumably replaced by the homogeneity and instrumentality of technological modalities of representation and presencing (Vesely, 2004; Perez-Gomez, 1983; Perez-Gomez and Pelletier, 1997). “There is no doubt that even the most advanced forms of representation are ultimately only tools, because they contribute to the representation of a given reality and only indirectly to its transformation. They are certainly not independent. They are more involved with, and reflect more clearly the conditions and limits of, our imagination and thinking than any earlier modes of representation” (Veseley, 2004). However, it is the premise here that mediated technologies and technique are not necessarily or simply “instrumental” and that they create an opening precisely due to their socalled technical nature (Heidegger, 1977; Ellul, 1965). The intent is that logos enables us to become more intelligent and adept with the technè of creating representations and of transforming the world. Mitchell and Thurtle (2004) refer to the “material poiesis of informatics [and to the] moments when information and flesh co-constitute one another. Understanding these moments allows us to illuminate the specific ways that we perform the transformation from the virtual world of information to the actual world of flesh and bone, as well as the implications of this moment for human biological existence (the species body), communal interactions (the social body), and the accessing and transformation of shared memory (the body of knowledge). [W]e have the capabilities to use informational technologies to generate new forms of fleshy experience.” Information technologies are not simply distinct modes of instrumental thought that overlay the body as simply their structural frame of reference. There is no one body and thus not one stable notion that can ground all else in the realm of artifice. “[Phenomenology] too often reifies the body as the ‘zero point’ of sense experience and does little to help us understand the integrity of mediated experience in linking and transforming bodily interaction. The cultural dimensions of mediated experience end up as a problem that begs explanation, rather than being understood as a reverberant and active part of embodied experience” (Mitchell and Thurtle, 2004). CIMSp assumes that the integration of new media technologies and information into the inherently embodied architectural design process requires an “informational management system” that makes the creator more intelligent and creative rather than the obverse as the charges of instrumentality claim. Furthermore, the potential and necessity of a support structure within the collaborative workflow is all the more critical with digital media versus more traditional (and solitary) modes of making. The larger project workflow is being simultaneously investigated and created to provide the infrastructure for technical and knowledge access in regard to procedures and software applications, documentation and feedback loops for project information and metadata, archival organization systems, and data management tactics to structure the production environment. The group is working with a group that specializes in courseware and XMLbased systems for the organization of a varied set of data, documents, files, and courseware (fig. 2). This system will work in conjunction with a commercially available content management application. CIMSp is a circuit of “information” (more of a game than a systematic methodology) that recognizes digital craft as embodied knowledge within the contingent and multifaceted design process. It continually makes apparent the collaborative nature of the creative act and requires a critical perspective on the information engaged. Although debatable, Rasch defines the notion of “information” as that which “has to be Figure 1: Infrastructural and procedural diagram of the CIMSp exemplifying the “circuitry” between the realms of mediation and making. The XML-driven content management system is a “permeable filter” that choreographs the temporal and spatial conditions within the infrastructure and workflow. distinguished from the notion of the message. Information is seen as the total field of choices from which the choice of correct message is to be made. Information is proportional to uncertainty: the greater the information, the greater the uncertainty on the part of the auditor regarding precisely which message from the possible manifold possible messages is the intended message” (Rasch, 2000). As such, the (digital) integration of various forms of information, from technical to cultural, into the creative process requires one to be ever increasingly more crafty and intelligent. VP Research; Dean Mahmoud; Dean Blockley, and Director Stephen Fai; the CIMS advisory board members; the Canadian Foundation for Innovation and the Ontario Innovation Trust. Lastly, I would like to express our gratitude to the Department of Canadian Heritage for granting the research funding. The project is funded through the Canadian Culture Online Program and supports the “Digital Architectural (Re)construction Program” within CIMS and establishes the unit as one of a handful of “New Media Research Network” nodes across Canada.